AirBnB Listings in Los Angeles, CA¶
Data source: http://insideairbnb.com/get-the-data.html¶
Date compiled: May 8, 2020¶
2 - Exploratory Data Analysis¶
Dependencies¶
In [1]:
from datetime import datetime
from scipy import stats
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px
from PIL import Image
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
from utils import utils, plotting
%matplotlib inline
In [2]:
# Settings
pd.options.display.max_columns = 200
pd.options.display.max_rows = 50000
sns.set(font_scale=1.2)
px.set_mapbox_access_token(utils.get_token())
Data¶
In [3]:
reviews = pd.read_csv('data/reviews.csv')
reviews.info()
In [4]:
listings = pd.read_pickle('data/listings_clean.pkl')
listings.info()
Rename and recast columns¶
In [5]:
# Rename columns to simpler names
listings.rename(columns={'host_is_superhost': 'by_superhost',
'neighbourhood_cleansed': 'neighborhood',
'review_scores_rating': 'score_rtg',
'review_scores_accuracy': 'score_accu',
'review_scores_cleanliness': 'score_clean',
'review_scores_checkin': 'score_check',
'review_scores_communication': 'score_comm',
'review_scores_location': 'score_loc',
'review_scores_value': 'score_val'}, inplace=True)
# Cast `by_superhost` as integer type
listings['by_superhost'] = listings.by_superhost.astype(int)
listings.head(2)
Out[5]:
Filter out listings with an outlier price¶
In [6]:
list765 = listings[listings.price <= 765].copy()
list765.describe()
Out[6]:
Is there a difference with listings by superhosts?¶
In [7]:
# Price distribution for listings by regular hosts and superhosts
plotting.hist_with_hue(df=list765, col='price', bins=range(0, 770, 10))
# Descriptive statistics of price
utils.desc_byhost(df=list765, col='price')
Out[7]:
In [8]:
# Rating distribution of listings by regular hosts and superhosts
plotting.hist_with_hue(df=list765, col='score_rtg', bins=32)
# Descriptive statistics of rating
utils.desc_byhost(df=list765, col='score_rtg')
Out[8]:
In [9]:
fig, ax = plt.subplots(1, 2, figsize=(16, 4))
# Rating distribution in relative terms
plotting.hist_pct(df=list765, ax=ax[0], filter_val=0, title='Regular Hosts', xlab='Rating Interval', ylab='Percent of Ratings')
plotting.hist_pct(df=list765, ax=ax[1], filter_val=1, title='Superhosts', xlab='Rating Interval')
In [10]:
# 2-sample t-test
reghost = list765[list765.by_superhost == 0].copy()
suphost = list765[list765.by_superhost == 1].copy()
stats.ttest_ind(reghost.score_rtg.dropna(), suphost.score_rtg.dropna(), equal_var=False)
Out[10]:
There doesn't seem to be a significant price difference between listings by regular hosts and listings by superhosts. The price distributions of the 2 groups follow a similar shape and the two means fall well within each others' standard deviations.¶
On the other hand, there is a significant difference in user rating with listings by superhosts. This is confirmed with the two-sample t-test performed above which yielded a t-statistic of -61.73. On average, superhost listings are rated 97.6 out of 100 +/- 3.3, compared to 92.2 +/- 10.7. This may not sound like much of a difference, but consider the count comparison in relative terms above. 97.4% of ratings on superhost listings were higher than 90/100, whereas only 71.3% of ratings on listings by regular hosts were higher than 90/100. This means that there is a greater chance of having a good experience if one chooses to stay at a property listed by a superhost. Whether (1) a listing gets better ratings because the host is a superhost or (2) the host became a superhost because they had high ratings is beside the point. The point is simply that superhost listings tend to have higher ratings.¶
The combination of these 2 things is a good sign. From the data, we can see that superhosts are offering a higher chance of a better experience at a similar price point on average.¶
Are higher ratings associated with higher costs?¶
In [11]:
# Price vs overall rating
sns.jointplot(data=list765, x='price', y='score_rtg');
In [12]:
score_vars = ['price', 'score_rtg', 'score_accu', 'score_clean', 'score_check', 'score_comm', 'score_loc', 'score_val']
# Price vs different rating types
plotting.scatter_subplots(df=list765, var_list=score_vars)
In [13]:
# Correlation
plotting.corr_heatmap(df=list765, var_list=score_vars, show_spearman=True, palette='viridis_r');
In [14]:
# Downsample high ratings
downsampled_rtg = utils.sample_byinterval(df=list765, col='score_rtg', lower=60, upper=100, step=10)
downsampled_rtg.score_rtg.hist()
Out[14]:
In [15]:
# Price vs overall rating
sns.jointplot(data=downsampled_rtg, x='price', y='score_rtg');
In [16]:
# Correlation of sample
plotting.corr_heatmap(df=downsampled_rtg, var_list=score_vars, show_spearman=True, palette='viridis_r');
The review variables all have a strong linear correlation with each other, which makes sense, but this is not what we are looking for. There is a very weak monotonic correlation between the price and the location score, and after downsampling the high ratings, between the price and the cleanliness score. But other than these weak correlations, there is no relationship between rating and price.¶
What are reviewers saying?¶
In [17]:
# Extract listings with an overall rating of less than 30
low = list765[list765.score_rtg < 30].id.tolist()
bad = reviews[reviews.listing_id.isin(low)].dropna(subset=['comments'])
len(low), len(bad)
Out[17]:
In [18]:
# Extract listings with an overall rating of 100
high = list765[list765.score_rtg == 100].id.tolist()
good = reviews[reviews.listing_id.isin(high)].dropna(subset=['comments'])
len(high), len(good)
Out[18]:
In [19]:
# Aggregate the bad reviews
bad_reviews = ' '.join(bad.comments.tolist())
# Aggregate the good reviews
good_reviews = ' '.join(good.comments.tolist())
# Approximate word count of each
len(bad_reviews.split(' ')), len(good_reviews.split(' '))
Out[19]:
In [20]:
plt.figure(figsize=(20, 10))
# Bad reviews
plt.subplot(1, 2, 1)
plotting.wordcloud(text=bad_reviews, bg_color='black')
# Good reviews
plt.subplot(1, 2, 2)
plotting.wordcloud(text=good_reviews)
In [21]:
# Stopword list
words = 'place stay Airbnb house apartment home day us booked said know staying something going wa Karla Jane John Ryan \
ha visit made unit one will even LA trip night didn got went time hour room please property manager Nickey \
another told week say someone go address thing people first though space hotel paid use book booking still \
pay later take friend hi Los Angeles Santa Monica cottage feel Hollywood make way well trying look came guest'
stopwords = set(STOPWORDS)
stopwords.update(words.split(' '))
plt.figure(figsize=(16, 16))
# Bad Reviews
plt.subplot(2, 1, 1)
plotting.wordcloud(text=bad_reviews, bg_color='black', stopwords=stopwords)
# Good Reviews
plt.subplot(2, 1, 2)
plotting.wordcloud(text=good_reviews, stopwords=stopwords)
Here, listings with an overall rating of less than 30/100 were collected and their reviews were treated as "bad" reviews. On the other hand, we have listings with an overall rating of 100/100 and their reviews were treated as "good" reviews. In generating a word cloud of good and bad reviews, it was my hope that we would be able to see what users enjoyed the most about their stays versus what users were complaining about the most. We cannot conclude anything based on these word clouds, but they did provide some general insights about good and bad listings.¶
Bad reviews: host being the biggest word here suggests the biggest complaint is about the host. We can see some of the other common complaints for bad listings, like roaches, the smell, the place being dirty, broken things, the host cancelling the reservation, or the listing being an automated posting.¶
Good reviews: the first thing to notice here is the size of the word "everything", which suggests a lot of the users that stayed at these properties enjoyed everything about their experience. A lot of reviews also include clean, spacious, amazing host, great location, walking distance, and convenient.¶
Common themes among the 2 groups are the host, the location, and of course the quality of the property.¶
Where are the best-rated listings?¶
In [22]:
# Extract listings with at least 50 reviews and an overall rating of 100
rating100 = list765[(list765.number_of_reviews >= 50) & (list765.score_rtg == 100)].copy()
# Calculate the average listing price of these listings by neighborhood
price100 = utils.agg_to_2cols(df=rating100, agg_col='price', agg_col_name='Price')
# Count the number of listings by neighborhood and select neighborhoods where the count is at least 10
ratings = utils.agg_to_2cols(df=rating100, agg_col='score_rtg', agg_col_name='Num_properties', agg_by_mean=False)
ratings = ratings[ratings.Num_properties >= 10]
# Neighborhoods with at least 10 listings that have at least 50 reviews and an overall rating of 100
plt.figure(figsize=(10, 5))
plotting.labeled_barplot(df=ratings, x='Num_properties', y='Neighborhood',
title='Neighborhoods with Maximum-rated Listings and Their Average Listing Price', upper=35)
# Show average price of each neighborhood
for i in range(11):
n = ratings.iloc[i, 1]
lab = price100.loc[price100.Neighborhood == ratings.loc[i, 'Neighborhood'], 'Price'].values[0]
plt.text(35.5, i, f'Avg price: ${lab:.02f}', va='center')
In [23]:
# Neighborhoods with at least 10 listings that have at least 50 reviews and an overall rating of 100
fig = plotting.choropleth_map(df=ratings, color_col='Num_properties')
fig.show()
Here, I defined best rated listings as listings with an overall rating of 100/100. But as a lot of these listings only had a few reviews, they're not as dependable so I filtered out those with less than 50 reviews. The bar plot and map above visualize the neighborhoods in LA that have at least 10 of these best-rated listings. Out of these 11 neighborhoods, Venice has the most at 29, followed by Silver Lake and Long Beach with 23 and 21 respectively.¶
Even though Long Beach is third on the list, it doesn't actually have that many if we take the area of the city into consideration, as seen on the map. We can see that the cluster of 3 adjacent neighborhoods - Venice, Santa Monica, and Mar Vista - have a total area that is much smaller than Long Beach, but combined have 57 best-rated listings. If we were looking for a good place to stay in LA, one of these 3 neighborhoods would definitely be a good place to start. This triad also has a wide range in price selections. Among these 3 neighborhoods, Mar Vista is the cheaper option with an average listing price of \$119.80 and Venice is the most expensive option with an average listing price of \\$232.45. Note that these averages only take into account the 100/100 listings (not all the other listings in those neighborhoods).¶
Where can we find low-cost versus high-end listings?¶
In [24]:
# Neighborhoods with at least 50 listings
nb50 = utils.agg_to_2cols(df=list765, agg_col='price', agg_col_name='Price', agg_by_mean=False)
nb50 = nb50[nb50.Price >= 50]
# Average listing price by neighborhood
prices = list765[list765.neighborhood.isin(nb50.Neighborhood)]
prices = utils.agg_to_2cols(df=prices, agg_col='price', agg_col_name='Avg_price')
# Average listing price of neighborhoods with at least 50 listings
fig = plotting.choropleth_map(df=prices, color_col='Avg_price', center_lat=34.20, center_lon=-118.25,
zoom=8.5, height=800, colorbar_range=(0, 300))
fig.show()
In [25]:
# Top and bottom listings in average price
plt.figure(figsize=(12, 6))
plotting.labeled_barplot(df=pd.concat([prices.iloc[:10], prices.iloc[-10:]]), x='Avg_price', y='Neighborhood',
title='10 Most Expensive and 10 Cheapest Neighborhoods', upper=400)
In [26]:
# 30 most expensive and 30 cheapest neighborhoods
fig = plotting.choropleth_map(df=pd.concat([prices.iloc[:30], prices.iloc[-30:]]), color_col='Avg_price', center_lat=34.20, center_lon=-118.25,
zoom=8.5, height=800, colorbar_range=(0, 300))
fig.show()
For the purpose of price comparison, I filtered out neighborhoods with less than 50 listings because there were neighborhoods with only a few listings and taking the average price of those neighborhoods wouldn't be as dependable for gauging the expected price of listings in the neighborhood. I chose 50 as the threshold because it cut out about half of the neighborhoods, leaving 130 remaining. This filtering alone was informative in its own way. Looking at the map with average prices, we can see huge gaps, especially around Downey and the neighborhoods directly below Palmdale. These gaps represent the neighborhoods that have less than 50 listings. These aren't necessarily bad places to look for a place to stay but now we know there are fewer options in these areas.¶
It looks like more than half of these 130 neighborhoods have an average listing price somewhere between \$100 and \\$150. The cheapest neighborhood is Historic South-Central with an average listing price of \$46.75 and the most expensive neighborhood is Malibu with an average listing price of \\$368.94. If we limit the map to the 30 most expensive and 30 cheapest neighborhoods, we can see a really interesting pattern. The most expensive neighborhoods (colored green - yellow) stretch along (almost) the entire coast of LA, going as far inland as Hollywood Hills at the northern coast. The cheapest neighborhoods are more central to eastern LA, as well as far north. What's interesting about both groups is that most of the neighborhoods in each group are connected, so for the most part, neighboring areas fall within a similar price range. In short, we should start our search inland if we're looking for low-cost listings or we can look along the coast if we're feeling a little luxurious!¶
Save the dataframe¶
In [27]:
listings.to_pickle('data/listings2.pkl')
pd.read_pickle('data/listings2.pkl').info()
In [28]:
listings.to_csv('data/listings2.csv')
pd.read_csv('data/listings2.csv').info()
In [ ]: